Skip to content

Background workers - #83

Merged
olivhoenen merged 45 commits into
iterorganization:developfrom
Yannicked:feature/celery-tasks
Jul 31, 2026
Merged

Background workers#83
olivhoenen merged 45 commits into
iterorganization:developfrom
Yannicked:feature/celery-tasks

Conversation

@Yannicked

Copy link
Copy Markdown
Collaborator

This PR introduces simulation ingestion via Celery background workers.

When simulations are uploaded through the new v1.3 API endpoint, file copying and status tracking are now handled by Celery tasks (copy_files_task chained with complete_ingestion_task) rather than blocking the HTTP request.

The ingestion pipeline tracks status though the following: QUEUED → COPYING → COPIED → VALIDATING → VALIDATED → COMPLETED, with failure variants.

@Yannicked
Yannicked force-pushed the feature/celery-tasks branch from 06bf048 to 68938a1 Compare May 28, 2026 08:19
@Yannicked
Yannicked requested a review from ioan-alexandra June 2, 2026 14:28
@Yannicked Yannicked mentioned this pull request Jun 8, 2026
@Yannicked
Yannicked marked this pull request as ready for review June 24, 2026 11:30
@olivhoenen

Copy link
Copy Markdown
Contributor

Please fix conflicts @Yannicked

@SimonPinches SimonPinches left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments:

  • I think you need to make celery a server dependency in pyproject.toml (or guard the import of celery) since otherwise pip install imas-simdb[server] will produce a server that crashes on import.
  • If the broker is down when apply_async() is called (the row is already committed as QUEUED), or a worker dies mid-COPYING (tasks are not acks_late, so the message is gone), the simulation will get stuck in a non-terminal state forever, and delete_simulation will return 409. The only recovery would seem to be manual DB surgery so we should probably add either an admin force-delete capability or a staleness timeout.
  • The watcher/notification behaviour is changed. Now body.add_watcher is ignored and the send_email_task never used - is this deliberate?

@Yannicked

Copy link
Copy Markdown
Collaborator Author
* I think you need to make `celery` a `server` dependency in `pyproject.toml` (or guard the import of `celery`) since otherwise `pip install imas-simdb[server]` will produce a server that crashes on import.

Done!

* If the broker is down when `apply_async()` is called (the row is already committed as QUEUED), or a worker dies mid-COPYING (tasks are not `acks_late`, so the message is gone), the simulation will get stuck in a non-terminal state forever, and `delete_simulation` will return 409. The only recovery would seem to be manual DB surgery so we should probably add either an admin `force-delete` capability or a staleness timeout.

Added the force-delete feature you suggested. On top of that: more graceful error handling (a failed apply_async() now marks the simulation COPY_FAILED and returns 503 instead of leaving it stuck), a cleanup task that marks copy/validation jobs stuck in a non-terminal state as failed, and a timeout on the tasks themselves. The task timeout defaults to 1h, for very datasets we might want to want to raise this

* The watcher/notification behaviour is changed. Now `body.add_watcher` is ignored and the `send_email_task` never used - is this deliberate?

Fixed!

@olivhoenen

Copy link
Copy Markdown
Contributor

I think it looks reasonably complete now, I propose we merge it now and continue the test with actual use-cases with #91, #97 and #94 when updated accordingly.

@olivhoenen
olivhoenen merged commit 7210cc6 into iterorganization:develop Jul 31, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Celery task queue

6 participants